For us humans, the idea of unclean data might seem a bit difficult to understand. If we are buying a product online and want to know the price, we would be able to understand it regardless of whether it was written as "$5", "5", "Five", etc. However, computers don't have this type of innate understanding. To them, "$5" is very different than "5" or "five". One of these reasons is that computers read information in types. Usually, these data types must not only be specified, but also uniform for a computer to understand them (to learn more about data types, what they are, and why they're important, visit this webpage).

Imagine you build a web scraper to collect images of french bulldog puppies and set it up so that it collects 1000 such images. While the majority of the images a well built web scraper returns will be of french bulldog puppies, some of the images it returns might be of other breeds or of french bulldogs that aren't puppies. Basically, some of the data will be irregular or irrelevant. If we disregard these irregularities and feed this data into our machine learning models to train it, the accuracy of the results will be skewed. In essence, our ai will lose "intelligence"; it will learn to less accurately spot a french bulldog puppy. This phenomenon may be easier to understand if we think of what it is like to teach a young child. If, from a young age, a child is taught by their parents that a "square" is an object with three angles, the child, not knowing any better, will grow up with the knowledge that a square is an object with three angles. Once the child grows and starts school, the accuracy of their knowledge will be tested. If the child is asked by the teacher to identify an object with three angles, they will incorrecly label it a "square".

The same happens with ai. Like a young child, it knows no information of the world and must be taught. This process of learning involves training and testing. We train ai by feeding it data and revealing to it the value of that data, much like how we would train a young child to identify a shape by showing it a shape and labeling it for them. We then test our ai by finding out the accuracy with which it executes its intended funcion, much like how we would test a child by asking them to identify a shape. This process of training and testing is repeated until we are satisfied with the accuracy of our ai.

Going back to our puppy image scraper example, had we taken the time to remove (clean) the irrelevant and inaccurate images once the web scraper collected them, we could have trained our ai to be accurate, useful, and very valuable. The proccess of the data cleaning, therefore, is crucial.

When we collected raw data from Amazon, we stored it in a file. The contents of a few of those files were available for you to see in our previous page. Is the data displayed in those files clean? No. The data in those files, while very understandable for a human, is not clean. How, then, would a clean data file look like? Let's take one of our already scraped files, use some data cleaning tools already built into Python (The programming language our web scraper is built with), and show what a clean data file would look like. To begin, take a look at the following image and notice the values of the data fields. Would you consider them clean?

The answer to the question above is no, the data cannot be considered clean. For two reasons: 1) most attributes have symbols (#, $, %) that don't allow for proper types and 2) the attributes with these symbols aren't in their proper types. You might think removing the symbols will solve the problem and you'd be somewhat correct and on the right track. However, there are two steps we have to take to properly clean this data: 1) remove the symbols, and 2) type cast the correct types we want the attributes to have.

The data we are working with requires a relatively simple process to clean. To remove the symbols, Pyhon has various functions we can use, such as the .strip() function, that removes unwanted characters. Check out a snippet of our code that not only makes use of this function to remove the percentage sign from our "5_Stars" attribute, but also type casts the result to an integer (the desired data type for this attribute).

In line 200, we used techniques previously discussed in the "web scraper" page to extract our "percent 5 star" attribute. After line 200, our attribute will return something like: "87%" with type "String". This is what you saw in the image of the table above; unclean data. Lines 201 and 202 remedy this. In line 201, we take our attribute and strip it of the "%" symbol. However, we want it to be of type "Integer", or "int", and it is currently type "String". Line 202 solves this issue with a simple "int" cast.

This process is repeated for a lot of the remaining attributes: "4_Stars", "3_Stars", "2_Stars", and "1_Star". Other attributes, such as the "Rating" attribute, use a similar process. For this attribute, we have to remove the words "out of 5 stars" (keeping only the information we want) and type cast it to a "float" (a decimal). To remove "out of 5 stars", we make use of another one of Python's useful functions, .split(). This function splits a string by a specified separator. If no separator is specified, as in our case, it will use any whitespace as the separator. Check out the code below.

Similar to the "percent 5 star" attribute, the first line of code (line 183) extracts the data from the Amazon product page. The following lines clean the data. Line 184 applies the .split() function to isolate the only information we are interested in and line 185 collects the item from the returned list. The list will roughly look like this: ["4.4", out", "of", "5", "stars"]. Index 0 has "4.4", index 1 has "out", and so on. The code in line 185 exists because the only information we are interested in is in index 0. Line 186 casts the data to a float, the type we need for this specific data. If you want to check out the rest of the code used to clean the remaining attributes, feel free to check our our Github repository.

Now that you have a better understanding of what the data cleaning process looks like, take a look at the same table as above, now occupied with data that has been cleaned. Since you cannot see the data types of the attributes, this information has been provided on the first row of items. Each column has uniform data types.

This concludes our analysis of the data cleaning process. Now that our data is clean, we can move onto machine learning. We have already covered how machines learn (the process of training and testing). In the following sections we will expand on the concepts already discussed and show some of the useful machine learning models that will allow us to analyze our data, find useful patterns, and extract valuable information. The next section will go into more detail about what machine learning actually is.

Supervised learning - The most common type of machine learning algorithm. This algorithm is used for a variety of tasks such as identifying faces in images, language translation, and product recommendations. In supervised learning, we give the machine an input (think of it as question) and the correct response (the answer). We call it "supervised" learning because of the nature of the learning process. Think of the machine learning algorithm as the student and us as the teacher. In the beginning, the student doesn't know anything about how to arrive at the proper conclusion so it just guesses at the answer. We then tell it the correct answer, how accurate is was, and the process repeats. Eventually, given we feed the algorithm enough quality data, it will begin to improve in the accuracy of its answers.

Unsupervised learning - In unsupervised learning, we feed data into the model and ask it to make sense of the structure. The algorithm will do this by identifying patterns in the data and finding similarities that allow for the data to be split into categories (usually done through a process known as clustering). The algorithm isn't designed to single out specific types of data, it will simply look for data that can be grouped by its similarities (and for outliers that stand out). An example of where unsupervised learing can be useful: An advertising platform segments the U.S. population into smaller groups with similar demographics and purchasing habits so that advertisers can reach their target market with relevant ads.

Reinforcement learning - Reinforcement learning involves giving the algorithm a reward based on whether it has done the desired action or not. The algorithm is designed to maximize the rewards it receives and will, with time and repetition, learn the ideal actions it must execute to receive the largest reward posible. Think of the way a dog is trained to sit; we reward it with a treat when it performs the desired action. With time, the dog will learn that if it sits when we ask it to, it will be rewarded.

That concludes our overview of machine learning. In the following section we will discuss the practical applications of machine learning, specifically what models we can use to help us visualize our data and analyze it through novel perspectives.

While there are a multitude of conventional ways to analyze data, our mission is to benefit from powerful machine learning models that analyze our data. Although we will also be using other visual models that we find practical for our particular dataset, the machine learning aspect will be our main goal. After all, it is this machine learning aspect that will allow our product to be scaled in such a way that it can serve as a unseful tool for a wide variety of businesses.

To begin with, let's import one of our scraped datasets and use some visual models to find useful patterns.

Use the following dropdown to select the model you want to view. We encourage you to analyze each model and find interesting patterns within our data. Each model will be accompanied by a few words to describe it and offer brief insight. Feel free to review this link which contains the raw dataset from which the models are based on. The data is very similar to the datasets available in the previous page. However, we have included some additional attributes.

{% if mode == 'heatmap' %}

Heatmap

The following model is a correlation heatmap. The correlations range from -1.0 to 1.0. A lighter colored square with a relatively high correlation value signifies a strong positive correlation between two attributes. This means that as the value of one attribute increases, the value of the other attribute also increases. Similarly, as the value of one attribute decreases, the value of the other attribute also decreases. For example, you will note that as a product received a higher number of five star reviews, its rating increased. Darker colors, on the other hand, signify that there is a negative correlation between attributes; as one value increases, the other decreases. Please note that only attributes with numeric values were included in the figure.

{% elif mode == 'scatter_1' %}

Positive Correlation (Scatterplot)

The following model is a scatterplot with an x-axis variable of "percent 5 stars" and a y-axis variable of "rating". Each dot on the graph represents a row (a product) on the dataset. Additional information displayed on the graph includes color hue of dots indicating range of answered questions and size of dots indicating range of rating counts. This type of graph helps us spot clusters and correlations within our data. Spotting correlations, however, is easier and more accurate on a Jointplot.

{% elif mode == 'scatter_2' %}

Negative Correlation (Scatterplot)

The following model is a scatterplot with an x-axis variable of "percent 1 star" and a y-axis variable of "rating". Each dot on the graph represents a row (a product) on the dataset. Additional information displayed on the graph includes color hue of dots indicating range of answered questions and size of dots indicating range of rating counts. This type of graph helps us spot clusters and correlations within our data. Spotting correlations, however, is easier and more accurate on a Jointplot.

{% elif mode == 'scatter_3' %}

No Correlation (Scatterplot)

The following model is a scatterplot with an x-axis variable of "days since first listed" and a y-axis variable of "rating". Each dot on the graph represents a row (a product) on the dataset. Additional information displayed on the graph includes color hue of dots indicating range of answered questions and size of dots indicating range of rating counts. This type of graph helps us spot clusters and correlations within our data. You'll notice this graph shows very low correlation between these two variables.

{% elif mode == 'joint_1' %}

Positive Correlation (Jointplot)

The following model is a jointplot with an x-axis variable of "percent 5 stars" and a y-axis variable of "rating". This graph, unlike our scatterplot, gives us a regression line which allows us to easily spot relationships between the two variables. We can clearly see a positive correlation here. Additionally, the graph shows the volume of data at each value on the respective axis.

{% elif mode == 'joint_2' %}

Negative Correlation (Jointplot)

The following model is a jointplot with an x-axis variable of "percent 1 star" and a y-axis variable of "rating". This graph, unlike our scatterplot, gives us a regression line which allows us to easily spot relationships between the two variables. We can clearly see a negative correlation here. Additionally, the graph shows the volume of data at each value on the respective axis.

{% elif mode == 'joint_3' %}

No Correlation (Jointplot)

The following model is a jointplot with an x-axis variable of "days since first listed" and a y-axis variable of "rating". This graph, unlike our scatterplot, gives us a regression line which allows us to easily spot relationships between the two variables. With these particular variables, no clear correlation is visible. While there is a slight negative correlation, it is not high enough to warrant the conclusion that a clear relationship exists between these two variables. Additionally, the graph shows the volume of data at each value on the respective axis.

{% elif mode == 'pie' %}

Pie Chart

A pie chart diviedes a circle into sections and allows us see the percentage of each section in reltion to the whole. For this pie chart, we decided to show the percentages of stars a particular product received. While this type of chart isn't extremely useful in representing data for analysis, it gives us a clearer picure of our data.

{% endif %}